End.
原

Nginx GoAccess安装与配置，线上7890端口没对外开，需要配置WebSocket服务

2022-04-07 22:36 所属：Nginx 阅读数：209 最后更新：2022-04-07 22:42

1、下载并安装Geoip

yum install ncurses-devel


cd /usr/local/app
wget https://github.com/maxmind/geoip-api-c/releases/download/v1.6.12/GeoIP-1.6.12.tar.gz
tar -xzvf GeoIP-1.6.12.tar.gz
cd GeoIP-1.6.12
./configure
make


2、安装goaccess

yum install goaccess

或者

推荐使用源码安装

cd /usr/local/appwget https://tar.goaccess.io/goaccess-1.5.5.tar.gz
tar -xzvf goaccess-1.5.5.tar.gz
cd goaccess-1.5.5/
./configure --enable-utf8
make
make install


3、配置GoAccess

3.1 下载脚本 nginx2goaccess.sh

nginx2goaccess.sh 脚本如下：

#!/bin/bash
#
# Convert from this:
#   http://nginx.org/en/docs/http/ngx_http_log_module.html
# To this:
#   https://goaccess.io/man
#
# Conversion table:
#   $time_local         %d:%t %^
#   $host               %v
#   $http_host          %v
#   $remote_addr        %h
#   $request_time       %T
#   $request_method     %m
#   $request_uri        %U
#   $server_protocol    %H
#   $request            %r
#   $status             %s
#   $body_bytes_sent    %b
#   $bytes_sent         %b
#   $http_referer       %R
#   $http_user_agent    %u
#
# Samples:
#
# log_format combined '$remote_addr - $remote_user [$time_local] '
# '"$request" $status $body_bytes_sent '
# '"$http_referer" "$http_user_agent"';
#   ./nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'
#
# log_format compression '$remote_addr - $remote_user [$time_local] '
# '"$request" $status $bytes_sent '
# '"$http_referer" "$http_user_agent" "$gzip_ratio"';
#   ./nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $bytes_sent "$http_referer" "$http_user_agent" "$gzip_ratio"'
#
# log_format main
# '$remote_addr\t$time_local\t$host\t$request\t$http_referer\t$http_x_mobile_group\t'
# 'Local:\t$status\t$body_bytes_sent\t$request_time\t'
# 'Proxy:\t$upstream_cache_status\t$upstream_status\t$upstream_response_length\t$upstream_response_time\t'
# 'Agent:\t$http_user_agent\t'
# 'Fwd:\t$http_x_forwarded_for';
#   ./nginx2goaccess.sh '$remote_addr\t$time_local\t$host\t$request\t$http_referer\t$http_x_mobile_group\tLocal:\t$status\t$body_bytes_sent\t$request_time\tProxy:\t$upstream_cache_status\t$upstream_status\t$upstream_response_length\t$upstream_response_time\tAgent:\t$http_user_agent\tFwd:\t$http_x_forwarded_for'
#
# log_format main
# '${time_local}\t${remote_addr}\t${host}\t${request_method}\t${request_uri}\t${server_protocol}\t'
# '${http_referer}\t${http_x_mobile_group}\t'
# 'Local:\t${status}\t*${connection}\t${body_bytes_sent}\t${request_time}\t'
# 'Proxy:\t${upstream_status}\t${upstream_cache_status}\t'
# '${upstream_response_length}\t${upstream_response_time}\t${uri}${log_args}\t'
# 'Agent:\t${http_user_agent}\t'
# 'Fwd:\t${http_x_forwarded_for}';
#   ./nginx2goaccess.sh '${time_local}\t${remote_addr}\t${host}\t${request_method}\t${request_uri}\t${server_protocol}\t${http_referer}\t${http_x_mobile_group}\tLocal:\t${status}\t*${connection}\t${body_bytes_sent}\t${request_time}\tProxy:\t${upstream_status}\t${upstream_cache_status}\t${upstream_response_length}\t${upstream_response_time}\t${uri}${log_args}\tAgent:\t${http_user_agent}\tFwd:\t${http_x_forwarded_for}'
#
# Author: Rogério Carvalho Schneider <stockrt@gmail.com>

# Params
log_format="$1"

# Usage
if [[ -z "$log_format" ]]; then
    echo "Usage: $0 '<log_format>'"
    exit 1
fi

# Variables map
conversion_table="time_local,%d:%t_%^
host,%v
http_host,%v
remote_addr,%h
request_time,%T
request_method,%m
request_uri,%U
server_protocol,%H
request,%r
status,%s
body_bytes_sent,%b
bytes_sent,%b
http_referer,%R
http_user_agent,%u"

# Conversion
for item in $conversion_table; do
    nginx_var=${item%%,*}
    goaccess_var=${item##*,}
    goaccess_var=${goaccess_var//_/ }
    log_format=${log_format//\$\{$nginx_var\}/$goaccess_var}
    log_format=${log_format//\$$nginx_var/$goaccess_var}
done
log_format=$(echo "$log_format" | sed 's/${[a-z_]*}/%^/g')
log_format=$(echo "$log_format" | sed 's/$[a-z_]*/%^/g')

# Config output
echo "
- Generated goaccess config:
time-format %T
date-format %d/%b/%Y
log_format $log_format
"

# EOF


赋予执行权限，chmod +x nginx2goaccess.sh

3.2 执行脚本

sh nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" [$http_x_forwarded_for] $request_length $request_time $host $request_body [$time_iso8601] $http_host $upstream_addr - $upstream_response_time "$request_uri" [$request_id]'

生成time-format、date-format、log-format

备注：执行脚本前，Nginx的项目需要配置好access日志记录。

3.2.1 nginx-http配置

log_format 配置记录access日志的格式

http {
    #其他配置...

    log_format weblog '$remote_addr - $remote_user [$time_local] "$request" '
         '$status $body_bytes_sent "$http_referer" '
         '"$http_user_agent" [$http_x_forwarded_for] $request_length '
         '$request_time $host $request_body [$time_iso8601] '
         '$http_host $upstream_addr - $upstream_response_time '
         '"$request_uri" [$request_id]';

}

3.2.2 server配置

http {
    #其他配置...    access_log /var/log/nginx/api_blog-access.log weblog;}

配置后，运营GoAccess，则采集分析/var/log/nginx/api_blog-access.log 日志文件的内容。


3.3 修改配置

goaccess 的配置文件 更改日志格式
vi /usr/local/etc/goaccess/goaccess.conf
将3.2生成的数据写入配置文件time-format、date-format、log-format。

3.4 生成报告html文件

/usr/local/app/goaccess-1.5.5/goaccess -f /var/log/nginx/api_blog_access.log -o /mnt/hg/Project/blogApi/App/View/nginx-report.html --real-time-html


3.5 命令说明

-f 指定nginx日志文件
-p 指定日志格式文件，默认 goaccess 的配置文件 /usr/local/etc/goaccess/goaccess.conf
-o 输出到指定html文件
--real-time-html 实时刷新
--ws-url 绑定一个域名
--daemonize 使 GoAccess 作为守护程序运行(仅在 --real-time-html 开启下有效)。
--port 指定服务使用的端口。GoAccess 默认使用端口 7890 作为 WebSocket 服务器。请确保此端口可用。

3.6 中文

LANG="zh_CN.UTF-8" bash -c "/usr/local/app/goaccess-1.5.5/goaccess -f /var/log/nginx/api_blog_access.log -o /mnt/hg/Project/blogApi/App/View/nginx-report.html --real-time-html"


3.7 配置Nginx

配置server对应3.4生成的html文件项目即可。

下面仅仅是示例。

server {
    server_name api.blog.net;

    root /mnt/hgfs/blogApi;
    index index.php index.html;

    access_log /var/log/nginx/api_blog-access.log weblog;

    location / {
        proxy_http_version 1.1;
        proxy_set_header Connection "keep-alive";
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header Url $proxy_add_x_forwarded_for;
        if (!-f $request_filename) {
             proxy_pass http://127.0.0.1:9025;
        }
    }
}


访问 api.blog.net/nginx，即可以打开。




4、线上配置WebSocket

4.1 nginx-http配置

http {
  # 省略....
  map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
  }

  upstream goaccess_blog_api {
    server 127.0.0.1:7890;
  }
}


4.2 server-配置，开启代理

location /goaccessBlogApi {
    proxy_pass http://goaccess_blog_api;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
}


4.3 启动access

主要配置参数 --ws-url
/usr/local/app/goaccess-1.5.5/goaccess -f /var/log/nginx/api_blog-access.log -o /data/project/blogApi/Public/Nginx/nginx-report.html --html-report-title=ApiBlog --real-time-html --daemonize --ws-url=api.blog.net/goaccessBlogApi --port=7890



5、读取多日志文件

zcat access.log.*.gz | goaccess access.log -
如：
zcat /var/log/nginx/api_blog-access.log-*.gz | /usr/local/app/goaccess-1.5.5/goaccess /var/log/nginx/api_blog-access.log - -o /data/project/blogApi/Public/Nginx/nginx-report.html --html-report-title=ApiBlog --real-time-html --daemonize --ws-url=api.blog.net/goaccessBlogApi --port=7890


End.